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EXECUTIVE  SUMMARY 


The  Link  16  data  format  was  designated  on  October  18,  1994,  as  the  Department  of  Defense  primary  data 
link  for  all  Service  and  Defense  Agency  C3I  and  weapon  system  applications.  In  preparation  for  the 
integration  of  Link  16  into  the  B-l,  the  B-l  System  Program  Office  requested  the  Crew  Station  Evaluation 
Facility  to  perform  a  series  of  human-in-the-loop  studies  evaluating  B-l  Link  16  system  human  interface. 


The  primary  objectives  of  this  evaluation  were  to  1)  establish  a  B-l  defensive  and  offensive  station  base 
line  performance  using  audio  tasking,  and  2)  compare  the  base  line  performance  with  performance  for  a 
freetext  message  and  an  integrated  mode.  This  study  looked  at  interface  issues,  not  specific  hardware. 
Two  part-tasks,  re-targeting  (Offensive  System  Officer  (OSO))  and  threat  updating  (Defensive  System 
Officer  (DSO)),  were  used.  Due  to  experimental  constraints,  only  two  modes,  audio  and  freetext,  were 
tested  for  the  DSO  task.  The  dependent  measures  for  both  tasks  were  key  strokes  per  target  or  threat,  time 
per  target  or  threat,  errors  per  target  or  threat,  and  Subjective  Workload  Analysis  Technique  (SWAT).  The 
OSO  and  DSO  tasks  were  analyzed  separately. 

The  repeated  measures  Multivariate  Analysis  of  Variance  (MANOVA)  for  the  OSO  task  showed  the 
integrated  mode  was  significantly  (p<0.05)  lower  for  number  of  strokes,  time,  errors  and  workload  than  the 
audio  or  freetext  modes.  Subjective  rank  ordering  of  the  three  modes  indicated  the  Weapon  System 
Officers  (WSOs)  unanimous  preference  for  the  integrated  mode. 

For  the  DSO  task,  the  MANOVAs  for  strokes  and  time  showed  significant  (p<0.05)  main  effects  for 
modes.  The  main  effects  for  error  and  workload  were  not  significant  (p<0.05).  The  number  of  strokes  was 
higher  for  the  freetext  compared  to  the  audio  mode,  but  the  time  to  accomplish  the  task  was  less.  Although 
more  strokes  were  required,  the  compactness  of  the  task  permitted  more  efficiency  and,  thus,  the  lower 
time. 

The  results  support  the  conclusion  that  the  integrated  mode  significantly  decreased  the  number  of  strokes, 
time,  and  errors  to  accomplish  the  retargeting  task,  and  was  the  preferred  mode  by  the  WSOs.  Also,  the 
lower  SWAT  scores  suggest  a  lower  workload  associated  with  the  integrated  mode.  The  integrated  mode  is 
recommended  for  the  OSO  and  DSO  stations. 
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1.  INTRODUCTION 


The  success  of  future  battlefield  operations  will  depend  on  how  efficiently  forces  attack  an  extremely 
complex  array  of  targets  while  at  the  same  time  countering  a  diverse  set  of  threats,  including  advanced 
offensive  information  warfare  capabilities.  Battlefield  success  will  be  achieved  through  the  integration  of 
ground  and  airborne  weapons  systems  via  digital  data  links.  Real-time,  high  capacity  data  transfer  between 
weapons  platforms  and  Command,  Control,  Communications,  Computers,  Intelligence,  Surveillance  and 
Reconnaissance  (C4ISR)  systems  are  required  for  more  effective  battle  management  as  the  density  of  the 
air  combat  environment  increases.  As  conditions  change  during  mission  execution,  both  air  and  ground 
crews  require  an  effective  capability  to  exploit  new  and  updated  information  from  off-board  sensors. 


Joint  Tactical  Information  Distribution  System  (JTIDS)  or  Link-16  (the  NATO  term  for  JTIDS)  is  the 
backbone  for  the  United  States  Navy  and  Air  Force  air  and  maritime  operations.  The  Link- 16  terminal 
implements  the  Tactical  Digital  Information  Link-J  (TADIL-J)  message  standard.  Its  architecture  provides 
a  common  communications  net  to  a  large  community  of  airborne  and  surface  elements  within  line-of-sight 
The  Link  16  data  format  was  designated  on  October  18,  1994,  as  the  Department  of  Defense  (DoD) 
primary  data  link  for  all  Service  and  Defense  Agency  C3I  and  weapon  system  applications. 

JTIDS  and  Multi-functional  Information  Distribution  System  (MIDS),  in  conjunction  with  a  host  system, 
support  the  exchange  of  joint  approved  Military  Standard  (MIL-STD)  6016  Tactical  Digital  Information 
Link  (TADIL)  J  messages.  JTIDS  and  MIDS,  in  addition  to  fixed  format  messages,  are  capable  of  freetext 
messages  and  variable  message  formats. 

In  preparation  for  the  integration  of  Link  16  into  the  B-l,  the  B-l  SPO  requested  the  CSEF  to  perform  a 
human-in-the-loop  study  evaluating  human  performance  differences  between  the  audio  tasking  of  the 
Weapon  System  Officer  (WSO)  and  his/her  tasking  using  the  Link  16  digital  message  system. 


1.1  TEST  OBJECTIVE 

This  is  the  first  in  a  series  of  studies  to  look  at  the  B-l  controls  and  displays  relative  to  implementing  the 
Link  16  in  the  B-l.  The  primary  objectives  of  this  evaluation  were  to  1)  establish  a  B-l  offensive  and 
defensive  station  base  line  performance  using  audio  tasking,  and  2)  compare  the  base  line  performance  with 
two  levels  of  data  link  performance.  The  base  line  condition  was  compared  to  1)  digital  freetext  message 
with  operator  manual  input,  and  2)  digital  data  link  message  integrated  into  the  B-1B  avionics  system.  This 
study  evaluated  the  B-l  data  link  human  interface;  it  did  not  evaluate  hardware  per  se. 
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2.  METHOD 


2.1  SUBJECTS 

Eleven  Weapon  System  Officers  (WSO)  participated  in  the  evaluation.  All  subjects  were  qualified  B-l 
WSOs  and  represented  a  random  sample.  Total  flying  hours  ranged  from  650  to  4000  hours  with  an 
average  of  2507  hours.  B-l  specific  flying  time  ranged  from  250  to  2250  hours  with  a  mean  of  1 170  hours. 
The  mean  age  was  34.9  years. 


2.2  APPARATUS 


2.2. 1  Crew  Station  Evaluation  Facility  (CSEF) 

The  Crew  Station  Evaluation  Facility  is  an  Air  Force  human-in-the-loop  simulation  facility  managed  and 
operated  by  the  Crew  Systems  Branch  (ASC/ENFC)  under  the  Flight  Systems  Engineering  Division 
(ASC/ENF).  ASC/ENF  is  part  of  the  Engineering  Directorate  (ASC/EN)  at  Wright-Patterson  AFB.  CSEF 
is  a  customer-funded  facility  that  supports  System  Program  Offices  in  their  acquisition  engineering  through 
crewmember  -  vehicle  interface  evaluations  using  human-in-the-loop  real-time  simulation.  Currently,  the 
CSEF  has  the  capability  to  perform  full  and  part  mission  simulations  for  a  variety  of  aircraft  including  the 
F-22,  F-16,  B-l,  KC-135  and  T-38. 


2.2.2  Test  Bed 

The  B-l  Engineering  Research  Simulator  (ERS)  is  supported  by  the  B-l  SPO  (ASC/YDE)  and  operated 
and  maintained  by  CSEF.  The  B-l  ERS  contains  the  pilot,  co-pilot,  offensive  and  defensive  stations.  The 
system  does  not  employ  a  motion  base  or  a  visual  system.  The  cockpit  controls  and  displays  are  currently 
configured  in  a  mixture  of  configurations.  The  forward  cockpit  is  between  Block  B  and  C,  the  Defensive 
station  is  Block  F  and  Offensive  station  is  Block  E. 


2.2.3  Voice  Recording  System 

A  Creative  Technology  Ltd.  Wave  Studio,  version  3.2. 1.0,  recorded  the  voice  messages  for  the  audio 
condition.  The  same  male  voice  recorded  all  messages.  The  voice  messages  simulated  an  AW  ACS 
controller  directing  the  OSO  to  update  the  target  information  or  the  DSO  to  update  the  threat  data.  The 
presentation  of  the  recorded  message  was  automatically  controlled  by  the  B-l  ERS  computer  system.  The 
audio  message  was  only  presented  once. 

In  order  to  assess  the  audible  intelligibility  of  the  recorded  messages  and  B-l  ERS  headsets,  the  system  was 
tested  using  word  lists  patterned  after  the  Modified  Rhyme  Test.  The  Air  Force  Research  Laboratory  at 
Wright-Patterson  AFB  generated  the  word  lists.  A  male  voice  recorded  two  separate  50-word  lists  for 
testing  purposes  (see  Appendix  A).  Six  subjects  were  tested  (three  subjects  per  word  list)  seated  in  the  B-l 
ERS  wearing  the  headsets  used  during  the  study’s  data  collection  sessions.  From  the  Human  Engineering 
Guide  to  Equipment  Design  (1972,  p.  174),  a  formula  (percent  correct  ={number  right  -  (number  wrong/5)] 
x  2)  was  used  to  score  the  multiple  choice  answer  sheets.  In  MIL-STD-1472E  (1996),  DOD  Design 
Criteria  Standard  for  Human  Engineering,  a  97%  speech  intelligibility  (SI)  for  voice  communication 
indicates  exceptionally  high  intelligibility.  The  results  of  our  SI  test  indicated  an  intelligibility  level  of 
98.8%. 
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2.2.4  Experimenter's  Console 


The  test  engineer  controlled  the  simulator  operation  and  selected  appropriate  test  parameters  such  as  test 
subject  number,  test  session  number,  etc.,  using  a  keyboard.  The  test  engineer  sat  behind  the  WSO  wearing 
a  headset  to  monitor  the  audio  voice  messages. 


2.3  TASKS 

Two  part-tasks  were  used  to  compare  the  performance  for  the  base  line  audio  condition  and  two  data  link 
configurations.  These  were:  1)  Retargeting  (OSO),  and  2)  Updating  threats  (DSO).  The  CSEF  modified 
the  Defensive  Station  Upgrade  Program  (DSUP)  mission  for  the  Defensive  System  Operator  (DSO)  task, 
and  modified  a  mission  from  the  B-l  Weapon  Flexibility  Study  for  the  Offensive  System  Operator  (OSO) 
task. 


2.4  TARGETS/THREATS 

Three  target  and  three  threat  groupings  were  used.  The  targets/threats  represented  a  continuum  of 
difficulty;  i.e.,  one  target  or  threat  for  the  easiest  condition,  a  grouping  of  3  targets  or  threats  for  the  middle 
condition,  and  5  targets  or  threats  for  the  most  difficult  condition.  Four  different  targets  and  four  different 
threat  sets  were  used  to  limit  the  number  of  times  the  subject  entered  the  same  target  or  threat  coordinates. 
For  example,  there  were  four  1 -target  packages,  four  3 -target  packages  and  four  5-target  packages.  The 
same  approach  applied  to  the  threat  groupings.  See  Appendix  B  for  target  and  threat  data. 


2.5  AUDIO  AND  FREETEXT  MESSAGE  FORMAT 

The  OSO  audio  and/or  freetext  messages  (FTM)  were  based  on  the  9-line  In-Flight  Target  Assignment 
(IT A)  format  (see  Table  1).  While  the  use  of  additives  (a  predetermined  value  added  by  the  crew  member 
to  the  transmitted  value)  is  operationally  approved  and  occasionally  implemented,  typically,  secure  voice  is 
available  and;  therefore,  additives  are  not  necessary.  Additives  were  not  used  during  this  study.  Only  the 
appropriate  lines  of  the  9-line  message,  as  defined  for  this  study,  were  used  (Items  #  1,  2,  3,  and  4).  The 
same  information  was  supplied  verbally  (recorded  message)  for  the  audio  condition.  For  example,  the 
voice  message  for  a  one-target  scenario  was:  “Strike  01.  Target  group  20.  Change  target  6.  New  target 
information  as  follows:  Item  1:  N3406.010  W11715.250,  01100.  Item  2:  1225-1235.  Item  3:  one.  Item  4: 
N3424.0  W1 1720.0.”  For  the  same  one-target  scenario  the  FTM  read: 

ALERT:  Target  group  20  Chg  target  6  New  target  information  as  follows: 

ITEM  1:  N3406.010  W11715.250  01100 
ITEM  2:  1225-1235 
ITEM  3:  1 

ITEM  4:  N3424.0  W1 1720.0 

A  different  format  was  used  for  the  DSO  messages.  As  an  example,  for  a  DSO  task  updating  one  threat, 
the  subject  heard  the  voice  message  “Strike  01.  Badger  active  N3435.5  W1 1745.5.”  For  the  same 
scenario,  the  freetext  message  read:  “ALERT:  SA7  N3435.5  W1 1745.5.”  Code  words  indicated  the  type  of 
threat;  i.e.,  Badger  was  the  code  word  for  a  SAM  7  missile. 

Voice  message  duration  for  the  OSO  task  ranged  from  43.49  seconds  to  127.23  seconds.  For  the  DSO  task, 
the  voice  message  duration  ranged  from  14.5  to  71  seconds.  The  average  message  length  for  both  tasks  by 
target  and  threat  grouping  is  presented  in  Table  2. 
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TABLE  1.  IN-FLIGHT  TARGET  ASSIGNMENT  FORMAT 


ITEM  1:  Target  Coordinates  (WGS-84),  Elevation 
ITEM  2:  Time  on  Target  (TOT) 

ITEM  3:  #  Weapons/Interval  (msec) 

ITEM  4:  Initial  Point  (IP) 

ITEM  5:  Offset  Aim  Point(s)  (OAP) 

ITEM  6:  Ingress 
ITEM  7:  Egress 
ITEM  8:  Remarks 

Authentication _ 


TABLE  2.  AVERAGE  MESSAGE  LENGT1 

4 

OSO  TASK 

AVERAGE 

DURATION  (Seconds) 

DSO  TASK 

AVERAGE 

DURATION  (Seconds) 

1  Target 

49.01 

1  Threat 

15.25 

3  Targets 

87.93 

3  Threats 

39.25 

5  Targets 

121.46 

5  Threats 

66.5 

2.6  DATA  LINK  CONTROLS/DISPLAYS 
2.6.1  OSO  Controls/Displays 

The  task-relevant  Multi-Function  Display  (MFD)  pages  were  modified  to  provide  a  data  link  capability  for 
the  study.  The  top  portion  of  the  MFD  page  displayed  the  data  link  free  form  text  message  (FTM).  When  a 
FTM  was  received,  the  words  “LINK  16”  flashed  at  the  bottom  of  the  MFD  page.  The  OSO  viewed  the 
FTM  with  one  button  push.  The  FTM  area  measured  4  lines  with  55  characters  per  line.  Navigation  data 
normally  displayed  in  this  area  was  still  available  on  the  second  OSO  MFD  from  the  navigation  page  (i.e., 
the  normal  page  displayed  on  the  right  MFD). 

For  this  study,  other  OSO  controls  were  modified  to  provide  a  means  to  display  the  FTM,  a  FTM  scroll 
capability,  and  a  control  to  enter  the  new  data  into  the  mission  plan  (integrated  condition).  The  FTM  area 
was  displayed  for  the  ffeetext  and  integrated  conditions.  In  addition,  for  the  integrated  condition,  a  data 
link  page  displayed  the  new  target  coordinates  (see  Appendix  C). 

For  the  integrated  condition,  the  OSO  received  an  FTM  alerting  him/her  to  new  target  coordinates.  The 
OSO  maneuvered  to  the  data  link  page.  The  OSO  viewed  the  FTM,  simultaneously,  with  the  Link  16  page 
to  verify  that  the  new  target  information  was  correct  on  the  Link  16  page.  To  enter  the  FTM  information 
automatically  into  the  mission  plan,  the  OSO  pushed  the  “accept”  button.  No  manual  data  entry  was 
required  for  the  integrated  condition. 
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r  TABLE  3.  CONTROLS/DISPLAYS 

CONTROL/DISPLAY 

FUNCTION 

OSO 

DSO 

Audio 

FTM 

Inte¬ 

grated 

FTM  View 

Displayed  the  FTM  on  the  top  portion  of 
the  MFD 

X 

X 

X 

X 

Scroll 

Scroll  the  FTM  up  and  down 

X 

X 

X 

X 

Start/Stop 

Start  and  stop  each  trial 

X 

X 

X 

X 

X 

Accept 

Automatically  input  the  new  target  data 
into  the  mission  plan 

X 

Link  16  page 

Displayed  in  action  point  table  format  the 
new  target  information 

X 

X 

Integrated  Keyboard 
(IKB) 

Entered  alphanumeric  data  and  maneuvered 
through  the  various  MFD  pages 

X 

X 

X 

X 

X 

MFD 

Displayed  relevant  function  page 

X 

X 

X 

X 

X 

2.6.2  DSO  Controls/Displays 

As  described  for  the  OSO,  a  similar  FTM  area  and  controls  were  provided  for  the  DSO  MFD,  specifically 
the  Defensive  Order  of  Battle  page  (see  Appendix  D).  The  DSO  data  link  controls/displays  were  only  used 
for  the  freetext  condition.  For  the  audio  condition  the  standard  MFD  page  (without  FTM)  was  displayed. 


2.7  PROCEDURE 

Two  groups  of  four  subjects  and  one  group  of  three  subjects  participated  in  the  study.  Subjects  were 
briefed  on  the  general  study  purpose,  study  particulars,  B-l  ERS  study-relevant  controls  and  displays,  and 
the  Subjective  Workload  Assessment  Technique  (SWAT). 

SWAT  has  two  phases:  scale  development  and  event  scoring.  The  scale  development  trains  the  subjects  on 
the  use  of  the  three  descriptors  (time  load,  mental  effort,  and  psychological  stress)  and  obtains  data  on  how 
each  of  these  descriptors  individually  affects  the  WSO’s  own  perception  of  workload.  This  data  was 
obtained  by  each  WSO  rank  ordering  27  SWAT  cards  to  establish  his/her  own  personal  SWAT  profile. 
The  event  scoring  was  the  verbal  data  collection  after  each  test  trial. 

After  completing  the  briefing  session,  each  subject  had  individual  hands-on  simulator  practice  time  to 
familiarize  himself  with  the  B-l  ERS  and  perform  several  practice  trials.  As  a  minimum,  all  subjects 
performed  5  practice  trials.  Subjects  were  encouraged  to  perform  as  many  practice  trials  as  necessary  to 
feel  comfortable  with  the  equipment  and  the  tasks.  Once  ail  subjects  had  completed  the  hands-on  practice 
session,  the  testing  session  began.  The  sequence  of  events  for  a  group  of  four  WSOs  is  depicted  in  Table 
4.  With  the  group  of  three  WSOs,  the  three  subjects  alternated  turns  in  the  simulator  instead  of  testing  by 
twos.  They  maintained  consistency  with  the  other  two  groups  of  four  by  completing  four  test  blocks  on  day 
1  and  6  test  blocks  on  day  2. 


TABLE  4.  SEQUENCE  OF  EVENTS 

DAY  1 

BRIEFING 

ALL 

INDIVIDUAL 

HANDS-ON 

PRACTICE 

LUNCH 

WSO  1-2 
TESTING 

WSO  3-4 
TESTING 

DAY  2 

WSO  1-2 
TESTING 

DEBRIEFING 

LUNCH 

WSO  3-4 
TESTING 

DEBRIEFING 
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At  the  beginning  of  a  trial  the  simulator  was  positioned  approximately  10  minutes  from  the  bombing  target 
area  between  1000  to  4000  feet  altitude  depending  on  the  scenario.  The  MFD  was  on  the  list  page  (menu 
page).  A  time  limit  of  10  minutes  was  imposed  on  all  test  trials.  If  the  subject  exceeded  the  time  limit,  the 
test  trial  automatically  ended.  Once  a  subject  completed  his  task,  he  indicated  its  completion  by  pressing  a 
stop  button.  The  test  engineer  recorded  the  subject’s  SWAT  scores  and  continued  on  to  the  next  test  trial. 

For  the  audio  condition,  the  subjects’  notes  of  the  audio  message;  i.e.,  new  target  or  threat  information, 
were  collected  for  later  use. 

2.8  DATA  COLLECTION  SESSION 

Two  subjects  alternated  time  in  the  simulator.  While  one  subject  tested,  tire  second  WSO  relaxed  in  a 
waiting  area.  After  each  block  of  6  trials,  the  subjects  rotated.  On  day  1,  all  subjects  had  a  test  session  in 
the  afternoon.  On  day  2,  two  subjects  tested  in  the  morning  and  two  subjects  tested  in  the  afternoon.  The 
exception  to  this  sequence  is  discussed  for  the  group  of  three  subjects  in  the  procedures  section.  Each 
subject  ran  a  complete  block  of  one  task  before  testing  on  the  second  task.  Prior  to  the  start  of  each  task  for 
the  first  time,  the  subject  performed  at  least  one  practice  trial  to  refresh  his  memoiy  on  the  upcoming  task. 


2.8.1  Objective  Data 

Performance  data  were  collected  for  time  to  complete  the  task,  number  of  errors  made,  and  the  number  of 
keystrokes.  For  analysis  purposes,  the  dependent  measures  were  computed  as  time  per  target/threat, 
number  of  errors  per  target/threat,  and  number  of  strokes  per  target/threat. 


2.8.2  Subjective  Data 

SWAT  data  were  collected  after  each  test  trial  and  a  debriefing  questionnaire  collecting  preference  data 
was  administered  after  the  subject’s  final  test  session. 

2.9  EXPERIMENTAL  DESIGN 

The  design  was  divided  into  two  separate  designs,  the  OSO  task  was  a  3  x  3  repeated  measures  Multivariate 
Analysis  of  Variance  (MANOVA)  and  the  DSO  task  was  a  2  x  3  repeated  measures  MANOVA. 


2.9.1  OSO  Task 

Four  trials  were  accomplished  for  each  condition  for  a  total  of  36  trials  per  subject.  The  order  of  the  mode 
of  presentation  and  the  targets  was  randomized.  The  presentation  mode  was  constrained  not  to  appear  more 
than  4-5  times  in  any  one  position  of  order  across  the  subjects.  The  36  trials  were  subdivided  into  6  blocks 
with  6  trials  in  each  block.  Each  block  represented  one  mode;  i.e.,  there  were  2  blocks  of  audio  trials,  2 
blocks  of  freetext  trials,  etc.  The  order  of  the  blocks  was  randomized  across  subjects  and  the  order  of  the 
target  groups  was  randomized  within  the  blocks.  The  order  of  presentation  of  the  tasks  was  also 
randomized;  i.e.,  half  the  subjects  were  randomly  assigned  to  test  the  OSO  task  before  the  DSO  task,  and 
vice  versa.  See  Appendix  E  for  the  order  of  presentation. 
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TABLE  5.  OSO  EXPERIMENTAL  DESIGN 


OSO  TASK 

RETARGETING 

WSO  1-12* 

MODE 

TARGETS 

_ 

AUDIO 

44  trials 

44  trials 

44  trials 

■ 

DATA  LINK 
FREETEXT 

44  trials 

44  trials 

44  trials 

1 

DATA  LINK 
INTEGRATED 

44  trials 

44  trials 

44  trials 

1 

Total 

396  trials 

u 

♦Note:  the  study  was  designed  for  12  subjects.  One  subject  dropped  out  without  prior  notice;  thus,  a 
replacement  was  not  available.  The  above  table  reflects  the  actual  number  of  trials  tested  for  1 1  subjects. 


2.9.2  DSO  Task 

Four  trials  were  accomplished  for  each  condition  for  a  total  of  24  trials  per  subject.  The  order  of  the  mode 
of  presentation  and  the  threat  groups  was  randomized.  The  presentation  mode  was  constrained  not  to 
appear  more  than  6  times  in  any  one  position  of  order.  The  24  trials  were  subdivided  into  4  blocks  with  6 
trials  in  each  block.  Each  block  represented  one  mode;  i.e.,  there  were  2  blocks  of  audio  trials,  and  2 
blocks  of  freetext  trials.  The  order  of  the  blocks  was  randomized  across  subjects  and  the  order  of  the  threat 
groups  was  randomized  within  the  blocks.  See  Appendix  E  for  the  order  of  presentation. 

A  DSO  integrated  condition  was  not  tested.  For  an  integrated  DSO  condition,  the  threat  update  would 
automatically  occur;  i.e.,  the  new  or  revised  threat  information  would  automatically  appear  on  the  Threat 
Situation  Format  (TSF).  No  DSO  input  would  be  required,  ergo;  we  did  not  test  this  condition. 


TABLE  6.  DSO  EXPERIMENTAL  DESIGN 


DSO  TASK 

UPDATING  THREATS 

WSO  1-12* 

MODE 

THREATS 

Threat  Group  2 

Threat  Group  3 

AUDIO 

44  trials 

44  trials 

44  trials 

DATA  LINK 
FREETEXT 

44  trials 

44  trials 

44  trials 

Total 

264  trials 

♦Note:  the  study  was  designed  for  12  subjects.  One  subject  dropped  out  without  prior  notice;  thus  a 
replacement  was  not  available.  The  above  table  reflects  the  actual  number  of  trials  tested  for  the  11 
subjects. 
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3.  RESULTS 


Through  the  use  of  several  descriptive  and  inferential  procedures  provided  by  SPSS  (Statistical  Package  for 
the  Social  Sciences,  1993)  release  6.0,  the  experimental  data  were  analyzed  separately  for  each  of  the  two 
tasks.  The  major  statistical  procedure  used  to  test  the  effect  of  differences  between  modes  on  the  four 
dependent  measures— strokes  per  target/threat,  time  per  target/threat,  errors  per  target/threat  and  SWAT— 
was  a  repeated  measures  Multivariate  Analysis  of  Variance  (MANOVA).  Repeated  measures  MANOVAs 
were  completed  in  a  two  step  process.  First,  the  overall  data  for  the  particular  task  under  all  target  or  threat 
conditions  were  analyzed.  If  the  SPSS  multivariate  criterion  tests  were  significant  at  the  alpha  .05  level, 
differences  between  modes  under  each  target  or  threat  condition  were  examined.  MANOVAs,  associated 
with  this  second  step,  provided  estimates  that  could  be  generalized  to  the  population. 


3.1  OSOTask 

Figure  1  displays  the  average  values  for  strokes,  time,  and  SWAT  for  all  subjects  across  all  replications 
(N=99).  Figure  2  displays  average  error.  As  shown,  the  integrated  mode  is  associated  decidedly  with  the 
lowest  values  for  all  of  the  dependent  measures.  In  particular,  note  that  error  is  zero  in  all  cases  in  the 
integrated  mode.  The  freetext  mode  is  lower  than  the  audio  mode  in  time  and  SWAT,  but  is  higher  in 
average  strokes  and  errors.  More  indices  of  central  tendency  and  dispersion,  including  minimum  and 
maximum  values,  are  shown  in  Appendix  F,  Table  1.  Median  values  are  graphically  displayed  in  Appendix 
F,  Figures  1  (strokes,  time,  and  SWAT)  and  2  (error).  Some  positive  skew  exists  in  the  data,  but  does  not 
impact  further  analysis. 


120 


audio  freetext  integrated 


MODE 

Figure  1.  Comparing  Modes  in  the  OSO  Taslc  Mean  Strokes.  Time  and  SWAT 
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Prior  to  testing  the  null  hypothesis  of  no  difference  between  the  modes,  the  dependent  measures  were 
correlated,  revealing  that  strokes,  time,  and  error  (S-T-E)  were  significantly  correlated  with  each  other 
(p<0.05),  while  SWAT  was  not  correlated  with  any  of  the  other  measures  (Appendix  F,  Table  2).  This 
required  a  “doubly  multivariate”  repeated  measures  analysis  investigating  strokes,  time,  and  errors 
simultaneously.  A  separate  analysis  considered  SWAT.  Both  repeated  measures  MANOVA  (Appendix  F, 
Tables  3  and  4,  for  S-T-E  and  SWAT,  respectively)  show  significant  (p<0.05)  multivariate  tests  for  the 
main  effects  of  mode  and  target,  as  well  as  for  their  interaction.  Given  the  significant  interactions,  further 
analysis  focused  on  simple  main  effects,  wherein  the  three  communication  modes  were  compared  within 
each  of  the  target  conditions.  Figures  3  through  8  display  the  average  values  for  the  dependent  measures. 
For  all  conditions,  the  integrated  mode  is  much  lower  in  value  than  the  other  two  modes.  Note  that  freetext 
remains  highest  in  mean  strokes  for  each  of  three  target  conditions,  just  as  in  the  overall  case,  but  its 
associated  mean  error  is  lower  than  that  associated  with  the  audio  mode  for  the  most  difficult  target 
condition.  Numerous  measures  of  central  tendency  and  dispersion  for  each  mode  by  target  condition  are 
shown  in  Appendix  F,  Table  5.  Also,  as  with  the  overall  case,  median  values  are  graphically  portrayed  in 
Figures  2  through  7,  Appendix  F.  Note  that  less  positive  skew  (as  demonstrated  by  the  difference  between 
mean  and  median)  on  each  of  the  dependent  measures  is  associated  with  the  high  target  condition  than  for 
the  other  target  conditions. 


TARGET  LOW  CONDITION 

160  , - 


AUDIO  FREETEXT  INTEGRATED 


MODE 

Figure  3.  Comparing  Modes  in  the  OSO  Task:  Mean  Strokes,  Time  and  SWAT 

- 


MEANS 


AUDIO  FREETEXT  INTEGRATED 

MODE 

Figure  4.  Comparing  Modes  in  the  OSO  Task:  Mean  Errors 


Note  that  low,  medium  and  high  target  conditions  equal  1  target,  3  targets,  and  5  targets. 
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AUDIO  FREETEXT  INTEGRATED 
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Figure  8.  Comparing  Modes  in  the  OSO  Task:  Mean  Errors 


Separately,  within  each  target  condition,  the  modes  were  compared  through  repeated  measures 
MANOVAs.  Again,  as  in  the  overall  condition,  strokes,  time,  and  error  were  analyzed,  simultaneously, 
and  SWAT  by  itself.  Statistics  derived  from  these  MANOVAs  resulted  in  Tables  7  through  9,  which 
display  an  association  measure  of  effect,  similar  to  correlation,  “eta-squared,”  for  the  low  to  high  target 
conditions,  respectively.  The  repeated  measures  “eta-squared”  is  derived  by  dividing  the  sum  of  squares 
for  the  effect  of  interest  by  the  sum  of  squares  for  error  added  to  the  sum  of  squares  for  the  effect  of 
interest.  The  effect  of  interest  is  either  of  the  two  orthogonal  contrasts,  freetext  minus  audio  (FT-A),  and 
integrated  minus  the  average  of  audio  plus  freetext  (I-avg  (A+T)).  These  contrasts  are  transformed 
dependent  variables  provided  by  the  MANOVA  procedure.  For  each  of  the  four  dependent  measures  in 
each  of  the  target  conditions,  the  strength  of  the  association  is  greatest  with  the  latter  contrast  than  the 
former.  For  example,  the  difference  between  integrated  and  the  other  two  modes  in  combination  accounts 
for  98.7%,  99.16%  and  99.1%  of  variance  in  strokes  for  the  low  to  high  target  conditions,  respectively, 
compared  to  31.17%,  58.61%  and  27.97%  for  the  difference  between  freetext  and  audio.  For  each  of  the 
contrasts,  strokes  and  time  account  for  more  of  the  variance  than  error  or  SWAT.  Included  with  the  sample 
eta-squared  values  are  estimates  for  the  degree  of  association  in  the  population.  A  formula  provided  in 
Bray  and  Maxwell  (1990,  p.37)  adjusts  the  sample  for  the  more  general  case.  Table  10  displays  the  values 
over  all  target  conditions. 

Aside  from  eta-squared  values  for  each  contrast,  the  SPSS  procedure  provides  parameter  estimates  for 
univariate  tests  of  mean  differences.  These  are  shown  in  Tables  11  through  13  for  the  low,  medium  and 
high  conditions,  respectively.  As  in  the  previous  tables,  the  difference  between  integrated  and  the  other 
two  modes  is  much  greater  than  the  difference  between  freetext  and  audio  for  all  four  dependent  measures 
under  all  target  conditions.  For  instance,  in  the  integrated  mode,  the  WSOs  took  less  time  than  in  the  other 
two  modes  combined;  73.48,  46.39  and  42.91  seconds  for  the  low  to  high  target  condition,  respectively.  In 
contrast,  in  the  freetext  mode,  the  WSOs  took  on  average  36.59,  24.17  and  21.87  seconds  less  than  in  the 
audio  mode.  Also,  the  differences  between  the  two  contrasts  are  stark  for  strokes  and  time  compared  to 
errors  and  SWAT;  the  disparity  for  strokes  and  time  being  greater  under  the  low  target  condition  than  under 
the  other  two.  Note  the  difference  in  strokes  and  error  between  freetext  and  audio  is  generally  positive, 
indicating  freetext  was  worse  than  audio.  In  order  to  generalize  beyond  the  sample,  95%  confidence 
intervals  around  the  mean  differences  are  provided.  Table  14  displays  values  over  all  target  conditions. 
Finally,  in  regard  to  the  OSO  task,  replications  and  an  experience  variable  were  found  not  to  be  significant 
(p>0.05)  in  the  main  effects  or  interactions  (with  mode). 
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TABLE  7.  COMPARING  CONTRAST  EFFECTS  IN  THE  OSO  TASK:  ETA-SQUARED 


*p<.05,  **p<.01 

A-audio;  FT-freetext; 
I-integrated 

TARGET  CONDITION 

LOW 

MEASURE 

STROKES 

TIME 

ERRORS 

SWAT 

CONTRAST 

CONTRAST 

CONTRAST 

CONTRAST 

FT -A 

I-avg (A+FT) 

FT-A  I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

SAMPLE 

.3117 

.9870** 

.8697**  .9523** 

.0107 

.4204* 

.0212 

.2838 

POPULATION  ESTIMATE 

.2405 

.9856 

.8563  !  .9474 

(-.0916) 

.3604 

(-.0800) 

.2097 

TABLE  8.  COMPARING  CONTRAST  EFFECTS  IN  THE  OSO  TASK:  ETA-SQUARED 


**p<.01 

A-audio;  FT-freetext; 
I-integrated 

TARGET  CONDITION 

MEDIUM 

MEASURE 

STROKES 

TIME 

ERRORS 

SWAT 

CONTRAST 

CONTRAST 

CONTRAST 

CONTRAST 

FT-A 

I-avg (A+FT) 

FT-A  ; I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

SAMPLE 

.5861** 

.9916** 

.8477**  !  .9371** 

.1421 

.2327 

.1196 

.5984** 

POPULATION  ESTIMATE 

.5433 

.9908 

.8319  .9306 

.0533 

.1534 

.0285 

.5569 

TABLE  9.  COMPARING  CONTRAST  EFFECTS  IN  THE  OSO  TASK:  ETA- SQUARED 


**p<. 01 

A-audio;  FT-freetext; 
I-integrated 

TARGET  CONDITION 

HIGH 

MEASURE 

STROKES 

TIME 

ERRORS 

SWAT 

CONTRAST 

CONTRAST 

CONTRAST 

CONTRAST 

FT-A 

I-avg (A+FT) 

FT-A  | I-avg (A+FT) 

FT-A 

I-avg(A+FT) 

FT-A 

I-avg (A+FT) 

SAMPLE 

.2797 

.9910** 

.8126**  j  .9650** 

.3062 

.2638 

.2537 

.6628** 

POPULATION  ESTIMATE 

.2052 

.9900 

.7932  |  .9613 

.2344 

.1876 

.1765 

.6279 

TABLE  10.  COMPARING  CONTRAST  EFFECTS  IN  THE  OSO  TASK:  ETA-SQUARED 


*p< . 05,  **p<.01 

A-audio;  FT-freetext; 
I-integrated 

TARGET  CONDITION 

OVERALL 

MEASURE 

STROKES 

TIME 

ERRORS 

SWAT 

CONTRAST 

CONTRAST 

CONTRAST 

CONTRAST 

FT-A 

I-avg (A+FT) 

FT-A  j I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

SAMPLE 

.5112** 

.9936** 

.8958**  j  .9651** 

.0095 

.3718* 

.1493 

.5828** 

POPULATION  ESTIMATE 

.4618 

.9929 

.8853  !  .9616 

(-.0907) 

.3083 

.0632 

.5406 
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TABLE  11.  COMPARING  CONTRAST  EFFECTS  IN  THE  OSO  TASK:  MEAN  DIFFERENCES 


**p<. 01 

A-audio;  FT-freetext; 
I-integrated 

TARGET  CONDITION 

LOW 

MEASURE 

STROKES 

TIME 

ERRORS 

SWAT 

CONTRAST 

CONTRAST 

CONTRAST 

CONTRAST 

FT -A  | I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

MEAN  DIFFERENCE 

4.5227 

-63.4429** 

-36.5935** 

-73.4795** 

.0455 

-.3864 

-1.8816 

-9.4159 

LOWER  LIMIT  951  C.I. 

-.2125 

-68.5790 

-46.5722 

-85.0647 

-.2626 

-.7060 

-10.8884 

-19.9555 

UPPER  LIMIT  95%  C.I. 

9.2579 

-58.3069 

-23.0490 

-61.8942 

.3535 

-.0667 

7.1248 

1.1237 

TABLE  12.  COMPARING  CONTRAST  EFFECTS  IN  THE  OSO  TASK:  MEAN  DIFFERENCES 


**p< . 01 

A-audio;  FT-freetext; 
I-integrated 

TARGET  CONDITION 

MEDIUM 

MEASURE 

STROKES 

TIME 

ERRORS 

SWAT 

CONTRAST 

CONTRAST 

CONTRAST 

CONTRAST 

FT-A 

I-avg (A+FT) 

FT-A 

- . 

I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

MEAN  DIFFERENCE 

4.6742** 

-40.3673** 

-24.1656** 

-46.3904** 

.0758 

-.1212 

-4.3841 

-19.6533** 

LOWER  LIMIT  95%  C.I. 

1.9068 

-42.9789 

-31.3843 

-54.8563 

-.0554 

-.2763 

-12.7643 

-30.9972 

UPPER  LIMIT  95%  C.I. 

7.4416 

-37.7556 

-16.9469 

-37.9245 

2.0690 

.0339 

3.9962 

8.3094 

TABLE  13.  COMPARING  CONTRAST  EFFECTS  IN  THE  OSO  TASK:  MEAN  DIFFERENCES 


**p<.01 

A-audio;  FT- free 
I-integrated 

TARGET  CONDITION 

t  ex  L  t 

HIGH 

MEASURE 

STROKES 

TIME 

ERRORS 

SWAT 

CONTRAST 

CONTRAST 

CONTRAST 

CONTRAST 

FT-A 

I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

FT-A 

I-avg  (A+FT) 

FT-A 

I-avg (A+FT) 

MEAN  DIFFERENCE 

3.0045 

-35.9339** 

-21.8674** 

-42.9121** 

-.0727 

-.0955 

-6.2204 

-27.1306** 

LOWER  LIMIT  95% 

C.I. 

-.3926 

-38.3498 

-29.2676 

-48.6749 

-.1499 

-.2078 

-13.7374 

-40.7657 

|  UPPER  LIMIT  95% 

C.I. 

6.4016 

-33.5181 

-14.4673 

-37.1493 

.0044 

.0169 

1.2965 

-13.4954 

TABLE  14.  COMPARING  CONTRAST  EFFECTS  IN  THE  OSO  TASK;  MEAN  DIFFERENCES 


*p<.05,  **p<.01 

A-audio;  FT-freetext; 
I-integrated 

TARGET  CONDITION 

OVERALL 

MEASURE 

STROKES 

TIME 

ERRORS 

SWAT 

CONTRAST 

CONTRAST 

CONTRAST 

CONTF 

(AST 

FT-A 

I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

FT-A 

I-avg (A+FT) 

MEAN  DI FFERENCE 

4.0670** 

-46.5820** 

-27.5410** 

-54.2614** 

.0162 

-.2010* 

-4.1619 

-18.7335** 

!  LOWER  LIMIT  95%  C.I. 

1.2649 

-49.2224 

-34.1582 

-61.5272 

-.1004 

-.3851 

-11.1634 

-29.9023 

UPPER  LIMIT  95%  C.I. 

6.8691 

-43.9416 

-20.9238 

-46.9955 

.1149 

-.0169 

2.8395 

-7.5647 
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3.2  DSO  Task 


Figure  9  displays  the  average  values  for  strokes,  time,  and  SWAT  for  all  subjects  across  all  replications 
(N=66).  Figure  10  displays  average  error.  As  in  the  OSO  task,  the  ffeetext  mode  is  lower  in  time  and 
SWAT  than  the  audio  mode  and  higher  in  strokes.  However,  unlike  the  OSO  task,  error  on  average  is 
lower  in  the  freetext  condition.  More  indices  of  central  tendency  and  dispersion  are  displayed  in  Appendix 
G,  Table  1.  Figures  1  (strokes,  time,  and  SWAT)  and  2  (error).  Appendix  G  show  some  positive  skew  in 
the  data  as  in  the  OSO  task,  but  does  not  impact  further  analysis. 
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AUDIO  FREETEXT 


MODE 

Figure  9.  Comparing  Modes  in  the  DSO  Task:  Mean  Strokes,  Time  and  SWAT 


MODE 

Figure  10.  Comparing  Modes  in  the  DSO  Task:  Mean  Errors 


Examination  of  the  correlations  among  die  dependent  measures  (Appendix  G,  Table  2)  showed  no 
significant  correlations  (p>0.05)  between  any  of  the  measures.  As  a  consequence,  each  dependent  measure 
was  analyzed  by  a  separate  repeated  measures  MANOVA  (Appendix  G,  Tables  3  through  6  for  strokes, 
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time,  error,  and  SWAT,  respectively).  The  MANOVAs  for  strokes  and  time  showed  significant  (p<0.05) 
main  effects  for  mode.  For  time,  the  interaction  between  mode  and  threat  was  significant  (p<0.05)  but  not 
for  strokes  (p>0.05).  The  main  effects  and  the  interaction  for  error  and  SWAT  were  not  significant 
(p>0.05)  with  the  exception  of  threat  (not  the  focus  here)  and  SWAT.  Consequently,  these  two  last 
dependent  measures  were  dropped  from  further  analysis.  Further  analysis  focused  on  simple  main  effects 
for  both  strokes  and  time,  even  though  the  interaction  for  strokes  was  not  significant,  given  that  such 
analysis  has  proved  useful.  Figures  1 1  through  13  display  the  mean  values  under  audio  and  freetext  for  the 
low  (1  threat),  medium  (3  threats)  and  high  (5  threats)  conditions,  respectively.  Strokes  remain  higher 
under  freetext  for  all  threat  conditions,  but  time  is  lower.  Note  that  under  the  most  difficult  threat 
condition,  the  differences  are  less  extreme  than  under  the  other  conditions.  Other  measures  of  central 
tendency  and  dispersion  are  shown  in  Appendix  G,  Table  7.  Figures  3  to  5  (Appendix  G)  display  the 
median  values  for  each  threat  condition.  Positive  skew  is  less  pronounced  compared  to  the  OSO  task. 


TARGET  LOW  CONDITION 

50  . - - - - 


AUDIO  FREETEXT 

MODE 

Figure  11.  Comparing  Modes  in  the  DSO  Task:  Mean  Strokes  and  Time 


THREAT  MEDIUM  CONDITION 


MODE 


Figure  12.  Comparing  Modes  in  the  DSO  Task:  Mean  Strokes  and  Time 


17 


THREAT  HIGH  CONDITION 


50 

45 

40 

35 

30 

25 

20 

15 


AUDIO  FREETEXT 


MEASURES 


■strokes 

■time 


MODE 

Figure  13.  Comparing  Modes  in  the  DSO  Task:  Mean  Strokes  and  Time 


Since  only  two  modes  were  tested  in  the  DSO  task,  there  is  only  one  contrast,  freetext  minus  audio.  The 
eta-squared  values  for  each  threat  condition  and  over  all  threat  conditions  are  presented  in  Table  15.  Note 
that  while  under  all  conditions  the  strength  of  the  associations  between  the  contrast  and  time  are 
comparable  to  those  in  the  OSO  task,  for  strokes  they  are  much  higher.  This  is  not,  however,  reflected  in 
the  mean  differences  in  strokes  between  the  two  modes  as  shown  in  Table  16.  Although  still  positive,  the 
magnitude  of  the  difference  is  less  than  in  the  OSO  task,  and  the  differences  in  time  are  also  less.  This  is 
due  to  the  compact  nature  of  the  DSO  task  requiring  much  less  time  and  strokes  to  perform  and,  hence,  less 
variance  in  the  data.  As  in  the  OSO  task,  replications  and  an  experience  variable  were  found  not  to  be 
significant  (p>0.05)  for  main  effects  or  interactions. 
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TABLE  15.  COMPARING  THE  CONTRAST  FREE  TEXT -AUDIO  IN  THE  DSO  TASK:  ETA-SQUARED 


**p<. 01 

THREAT  CONDITION 

LOW 

MEDIUM 

HIGH 

OVERALL 

MEASURE 

MEASURE 

MEASURE 

MEASURE 

STROKES 

TIME 

- 

STROKES 

TIME 

STROKES 

TIME 

STROKES 

TIME 

SAMPLE 

.7100** 

. 9570** 

.6708** 

.8267** 

.7240** 

.8982** 

.8425** 

.9441** 

POPULATION  ESTIMATE 

.6795 

.  9524 

.  6361 

.8085 

.  6950 

.8875 

.8265 

.  9384 

TABLE  16.  COMPARING  THE  CONTRAST  FREE  TEXT -AUDIO  IN  THE  DSO  TASK:  MEAN  DIFFERENCES 


**p<.01 

THREAT  CONDITION 

LOW 

MEDIUM 

HIGH 

OVERALL 

MEASURE 

_ 

MEASURE 

MEASURE 

MEASURE 

STROKES 

TIME 

STROKES  j  TIME 

1  . - . - 

STROKES 

TIME 

STROKES 

TIME 

MEAN  DIFFERENCE 

2.1818** 

-15.3411** 

3.3207**|  -9.3272** 

3.7363** 

-10.7900** 

3.0795** 

-11.8190** 

LOWER  LIMIT  95%  C.I. 

1.9940 

-17.6335 

1.6814  j  -12.3358 

2.1110 

-13.3497 

2.1414 

-13.8453 

UPPER  LIMIT  95%  C.I. 

3.1643 

-13.0486 
i _ 

4.9600  1  -6.3187 

:  i 

5.3617 

-8.2303 

4.0176 

-9.7926 
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3.3  Subjective  Data 


In  the  post  test  questionnaire,  the  WSOs  rank  ordered  the  three  modes  (audio,  freetext,  integrated)  based  on 
their  preference,  i.e.,  the  mode  they  liked  best,  the  mode  easiest  to  use,  and  the  mode  least  likely  to  make 
errors.  The  integrated  mode  was  ranked  number  one,  unanimously,  as  the  preferred  method,  followed  by 
the  freetext  mode.  The  audio  mode  was  ranked  third.  For  the  mode  least  likely  to  make  errors,  integrated 
was  ranked  first,  followed  by  freetext,  and  audio  was  the  most  likely  to  make  errors.  Ten  out  of  11  WSOs 
ranked  the  integrated  mode  as  the  easiest  to  use,  followed  by  the  freetext  mode,  and  the  audio  mode  was 
ranked  last.  One  WSO  ranked  the  integrated  mode  easiest  to  use,  followed  by  the  audio  mode,  then  the 
freetext  mode. 


Use 


■  Integrated 

■  Freetext 
□  Audio 


Figure  14.  Rank  Order  Mode  for  Preference,  Ease  of  Use  and  Error  Free 


The  WSOs  also  rank  ordered  the  two  modes  for  the  DSO  task  (audio  and  freetext).  They  unanimously 
ranked  the  freetext  mode  as  tire  number  one  choice  in  terms  of  preference,  ease  of  use,  and  least  likely  to 
make  errors. 

The  Link  16  page  designed  to  implement  the  integrated  mode  was  moderately  acceptable.  Eight  of  the  1 1 
WSOs  responded  to  the  question  “Did  they  like  the  Link  16  page”  more  in  a  positive  direction  (4  slightly 
agreed,  2  moderately  agreed  and  2  strongly  agreed).  Three  of  the  WSOs  didn’t  like  it.  A  suggestion  to 
improve  the  page  was  to  show  only  the  data  that  was  being  changed,  i.e.,  targets  that  were  not  a  part  of  the 
FTM  were  included.  Another  suggestion  was  to  provide  the  capability  to  accept  or  reject  parts  of  the  Link 
16  message.  There  seemed  to  be  some  confusion  as  to  what  the  “Link  16  page”  was.  Some  of  the  related 
comments  made  suggestions  for  improving  the  FTM  format  thus  suggesting  that  some  WSOs  were 
confusing  the  FTM  area  with  the  Link  16  page.  Suggestions  for  improving  the  FTM  format  included 
numbering  the  target  coordinates  on  the  left  for  easier  cross-check,  and  to  display  the  whole  message 
without  needing  to  scroll. 

To  alert  the  WSO  to  the  presence  of  a  data  link  FTM,  the  words  “Link  16”  flashed  at  the  bottom  of  the 
MFD  page.  Eight  of  1 1  WSOs  said  the  flashing  message  was  sufficient  to  get  their  attention.  However,  it 
was  suggested  that  it  should  be  tied  into  the  B-l  master  caution  system  or  CITS.  Other  suggestions  were  to 
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use  bold  or  reverse  video  instead  of  flashing.  One  WSO  preferred  that  the  flashing  be  stopped  once  the 
FTM  was  displayed — he  found  the  flashing  to  be  distracting. 

The  questionnaire  highlighted  several  data  link  design  considerations  for  future  designs: 

1)  Don’t  use  prime  data  area  to  display  the  FTM 

2)  Use  a  separate  full  screen  capability  for  the  FTM 

3)  Display  the  FTM  in  its  entirety  so  scrolling  isn’t  required 

4)  Integrate  the  FIM  controls  into  the  system. 

The  WSO’s  recommendations  for  integrating  Link  16  into  the  B-l  were: 

1)  The  defensive,  offensive  and  pilot  stations  all  need  Link  16  data 

2)  Replace  existing  MFDs/EDUs/VSDs  with  Liquid  Crystal  Displays  (LCD)  to  provide  a  textual 
and  graphics  capability 

3)  Provide  a  beyond-line-of-sight  data  link  capability 

4)  Replace  the  integrated  keyboard  (IKB)  with  a  full  alphanumeric  keyboard 

5)  Integrate  Link  16  into  the  B-l  from  an  Air  Force-perspective,  rather  than  as  a  B-l  only 
solution. 
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4.  DISCUSSION 


The  primary  purpose  of  this  study  was  to  evaluate  the  difference  between  the  communication  modes  for 
WSO  part  tasks — three  modes  for  the  OSO  task  and  two  for  the  DSO  task.  The  major  tool  used  for  this 
evaluation,  repeated  measures  MANOVA  transformed  the  four  dependent  measures:  strokes  per  target  or 
threat,  time  per  target  or  threat,  error  per  target  or  threat,  and  SWAT  into  contrasts  or  difference  measures. 
These  contrasts  were  freetext  minus  audio  and  integrated  minus  the  average  of  the  other  two  combined. 
For  the  OSO  task,  overall  statistics  for  eta-squared  and  mean  difference  (derived  from  evaluating  both 
contrasts)  emphatically  supported  the  integrated  mode  as  the  best  environment  in  which  the  WSO  can 
work.  Eta-squared,  a  type  of  correlation  between  the  difference  measure  and  the  original  dependent 
measure(s),  showed  that  the  difference  between  the  integrated  mode  and  the  average  of  the  other  two 
modes  accounted  for  over  99%  of  the  variance  in  strokes,  more  than  96%  of  the  variance  in  time,  37%  in 
errors,  and  58%  in  SWAT.  The  corresponding  percentages  for  freetext  minus  audio  were  51%  in  strokes, 
89.5%  in  time,  less  than  1  %  in  errors,  and  less  than  15%  in  SWAT.  By  itself,  eta-squared  did  not  indicate 
the  direction  of  the  difference.  The  other  statistic,  mean  difference,  based  on  the  univariate  parameter 
estimates  for  each  contrast  favored  the  integrated  mode  over  the  other  two.  The  difference  between  the 
integrated  mode  and  the  other  two  modes  was  over  1 1  times  fewer  strokes  than  the  difference  between 
freetext  and  audio.  In  fact,  the  freetext  mode  exhibited  more  strokes  on  average  than  in  the  audio  mode. 
The  time  difference  was  twice  as  large  when  considering  the  integrated  contrast  versus  the  freetext/audio 
contrast.  Integrated  also  showed  12  1/2  times  less  error  (freetext  and  audio  were  about  even  in  error 
overall)  and  a  4  1/2  times  better  SWAT  rating. 

Because  of  the  interaction  between  mode  and  target,  the  focus  of  the  analysis  shifted  to  comparing  the 
modes  within  each  target  condition.  The  strength  of  the  association  for  the  integrated  contrast  was 
consistently  higher  for  each  target  condition  when  considering  time  and  strokes.  On  the  other  hand,  the 
effect  of  the  difference  between  freetext  and  audio  on  strokes  was  only  significant  in  the  medium  target 
condition,  being  about  twice  as  strong  as  the  low  and  high  conditions.  In  addition,  the  effect  on  errors  and 
on  SWAT,  although  not  significant,  was  several  times  greater  in  the  high  target  condition,  as  opposed  to  the 
other  two.  The  integrated  contrast  exhibited  much  greater  consistency.  This  suggests  an  instability  in 
measuring  differences  between  freetext  and  audio  that  could  be  affected  by  subtle  differences  in  an 
experimental  environment  or  real  world  milieu. 

The  results  for  the  integrated  mode  are  more  reliable  with  perhaps  greater  generalizability  to  the  WSO 
population  than  the  freetext/audio  modes.  Admittedly,  some  of  this  could  be  attributable  to  an  artifact  of 
the  present  study;  i.e.,  the  high  target  condition  necessitated  a  longer  FTM.  Since  the  message  area  was 
limited  to  displaying  only  four  lines  simultaneously,  the  WSOs  by  necessity  had  to  scroll  the  message  up 
and  down  for  both  data  entry  and  cross-check.  This  explains  the  increased  number  of  strokes  and  would 
seem  to  have  contributed  to  the  increase  in  errors.  This  explanation  is  also  supported  in  part  by  the 
subjective  results;  i.e.,  many  WSOs  stated  their  preference  to  see  the  message  in  its  entirety. 

Examining  the  mean  differences  demonstrated  that  the  differences  between  contrasts  held  up  for  each  target 
condition,  namely,  the  difference  between  integrated  and  the  other  two  modes  being  much  greater  than  the 
difference  between  freetext  and  audio.  However,  interestingly,  while  within  each  contrast,  the  differences 
in  SWAT  ratings  increased  as  would  be  expected  when  going  from  the  low  to  high  difficulty  condition;  just 
the  opposite  occurred  for  strokes,  time  and  for  the  most  part,  errors.  In  other  words,  as  evidenced  by  the 
general  descriptive  statistics,  SWAT  ratings  became  worse,  as  expected,  but  the  number  of  strokes,  amount 
of  time,  and  number  of  errors  decreased  from  the  low  to  high  conditions.  This  cannot  be  attributable  to 
position  effects— each  condition  appeared  in  each  position  an  equal  number  of  times  and  the  replications 
variable  was  not  significant  (p>0.05).  There  was  less  skew;  i.e.,  less  extreme  scores  in  the  higher 
conditions,  perhaps  making  the  most  difficult  condition  more  representative  of  the  greater  population.  In 
any  event,  the  results  support  superior  performance  in  the  integrated  mode.  This  performance  was 
reinforced  by  the  subjective  data;  i.e.,  the  WSOs  rank  ordered  the  integrated  mode  as  number  one  in  terms 
of  their  preference,  for  ease  of  use,  and  as  least  likely  to  make  errors. 
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The  DSO  results  were  less  definitive.  The  differences  between  the  contrasts  were  decidedly  less  than  in  the 
OSO  task.  This  is  attributed  to  the  efficiency  of  the  DSO  task.  All  the  data  entry  was  accomplished  on  one 
page,  the  DOB  page.  The  most  difficult  threat  condition  required  more  strokes  for  data  entry  but  the 
organization  of  the  DOB  page  makes  for  a  relatively  efficient  method  of  data  entry.  Whether  manually 
entering  one  set  of  threat  coordinates,  or  five  sets  of  threat  coordinates,  the  task  merely  required  entering 
more  of  like  data  on  the  same  page.  The  freetext  mode  compared  to  audio  mode  was  significantly  lower  in 
time  but  not  for  number  of  strokes.  The  number  of  errors  and  workload  were  not  significantly  different 
between  the  two  modes. 

Comparing  the  two  tasks,  the  OSO  task  was  much  more  complex  and  difficult  than  the  DSO  task.  For 
example,  between  the  two  tasks,  in  the  audio  and  freetext  modes  overall,  the  number  of  strokes  and  amount 
of  time  is  more  than  doubled.  Therefore,  designing  for  the  worst  case,  the  biggest  payoff  will  come  with 
implementing  the  integrated  mode  for  the  OSO  task;  this  task  being  the  more  onerous  of  the  two. 
However,  since  the  WSO  is  expected  to  function  as  both  the  OSO  and  DSO,  equally  sharing  time  between 
the  two  tacks  the  type  of  data  link  interface  implementation  should  be  consistent  for  both  the  offensive  and 
defensive  stations. 
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5.  CONCLUSIONS/RECOMMENDATIONS 


The  OSO  data  results,  both  objective  and  subjective,  clearly  demonstrate  the  superiority  of  the  integrated 
mode  as  compared  to  the  audio  and  freetext  modes.  In  all  areas,  number  of  strokes,  time,  number  of  errors, 
and  workload,  the  integrated  mode  was  significantly  better. 

The  DSO  task  results  were  not  as  consistent  as  the  OSO  results.  The  freetext  mode,  compared  to  audio 
mode,  was  significantly  lower  in  time,  but  not  for  number  of  strokes.  This  effect  was  attributed  to  the 
compactness  of  the  task.  The  number  of  errors  and  workload  were  not  significantly  different  between  the 
two  modes. 


The  subjective  data  highlighted  several  design  considerations.  For  the  Link  16  page  (automated  the  OSO 
retargeting  task),  suggestions  for  improving  the  page  design  were  to  show  only  the  data  being  changed, 
provide  a  capability  to  accept  or  reject  parts  of  the  freetext  message,  and  stop  flashing  the  alert  message 
once  the  freetext  message  was  acknowledged.  Future  data  link  design  considerations  are:  1)  Don’t  use 
prime  data  areas  to  display  FTM,  2)  Use  a  separate  full  screen  capability  for  the  FTM,  3)  Display  the  FTM 
in  its  entirety  so  scrolling  isn’t  necessary,  and  4)  Integrate  the  FTM  controls  into  the  system. 

The  WSOs  recommendations  for  integrating  Link  16  into  the  B-l  are: 

1)  The  defensive,  offensive  and  pilot  stations  all  need  Link  16  data 

2)  Replace  existing  MFDs/EDUs/VSDs  with  Liquid  Crystal  Displays  (LCD)  to  provide  a  textual 
and  graphics  capability 

3)  Provide  a  beyond-line-of-sight  data  link  capability 

4)  Replace  the  integrated  keyboard  (IKB)  with  a  full  alphanumeric  keyboard 

5)  Integrate  Link  16  into  the  B-l  from  an  Air  Force-perspective  rather  than  as  a  B-l  only  solution. 

Since  the  OSO  task  is  clearly  more  complex  and  tire  more  difficult  of  the  two  tasks,  the  offensive  station  is 
primary  in  shaping  recommendations  for  a  B-l  data  link  interface  design.  Designing  for  the  worst  case,  the 
biggest  payoff  will  come  with  implementing  the  integrated  mode  for  the  OSO  task.  The  type  of  data  link 
implementation  should  be  consistent  for  both  the  offensive  and  defensive  stations.  Therefore,  based  on 
these  study  results,  the  integrated  mode  is  recommended  as  the  most  beneficial  design  from  a  performance 
and  user  acceptance  perspective. 

Future  studies  should  address  the  design  details  for  an  integrated  approach.  This  study  evaluated  interface 
issues;  it  was  not  a  hardware  evaluation.  The  post-test  questionnaire  clearly  stated  the  WSOs  areas  of 
concern  with  this  study’s  interface  design  and  provided  some  recommendations  for  what  they  would  like  to 
see  in  a  future  data  link  hardware  design. 
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APPENDIX  A 


INTELLIGIBILITY  WORD  LISTS 


SHEET  5B 


1.  seek 

26.  math 

2.  din 

27.  nest 

3.  kill 

28.  took 

4.  beat 

29.  case 

5.  fed 

30.  bark 

6.  tip 

31.  tent 

7.  pin 

32.  not 

8.  ban 

33.  sup 

9.  beat 

34.  run 

10.  gang 

35.  hop 

11.  came 

36.  tear 

12.  day 

37.  sin 

13.  tang 

38.  bit 

14.  wick 

39.  cut 

15.  peace 

40.  bill 

16.  rig 

41.  cold 

17.  foil 

42.  win 

18.  puff 

43.  heap 

19.  rust 

44.  page 

20.  safe 

45.  peel 

21.  path 

46.  buck 

22.  sass 

47.  ten 

23.  fin 

48.  sale 

24.  lake 

49.  raze 

25.  paw 

50.  dud 

Generated  by  Air  Force  Research  Laboratory, 
Wright-Patterson  AFB 
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APPENDIX  A 


INTELLIGIBILITY  WORD  LIST 

SHEET  6B 


1. 

kin 

2. 

dip 

3. 

heave 

4. 

beat 

5. 

peach 

6. 

bad 

7. 

lake 

8. 

lark 

9. 

sun 

10. 

seed 

11. 

went 

12. 

cave 

13. 

cud 

14. 

vest 

15. 

hang 

16. 

tin 

17. 

pay 

18. 

dill 

19. 

sit 

20. 

kit 

21. 

pace 

22. 

pale 

23. 

sum 

24. 

beach 

25. 

rate 

26.  pen 

27.  teak 

28.  wed 

29.  tan 

30.  told 

31.  shop 

32.  sane 

33.  not 

34.  sass 

35.  pack 

36.  pub 

37.  keel 

38.  mat 

39.  dun 

40.  will 

41.  bug 

42.  book 

43.  came 

44.  fib 

45.  sick 

46.  just 

47.  pig 

48.  pip 

49.  raw 

50.  boil 
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APPENDIX  B 


Target  Data 


MISSION 

MSN  REF.  # 

TARGET  GROUP 

DIFFICULTY  LEVEL 

A 

#15 

30.9 

1  -  1  TARGET 

2- 3  TARGETS 

3- 5  TARGETS 

B 

#14 

20.9 

1  -  1  TARGET 

2  -  3  TARGETS 

3 -5  TARGETS 

C 

#32 

60.9 

1  -  1  TARGET 

2-3  TARGETS 

3  -  5  TARGETS 

D 

#12 

30.9 

1  -  1  TARGET 

2- 3  TARGETS 

3  -  5  TARGETS 

Threat  Data 


MISSION 

MISSION  # 

THREAT  GROUP 

DIFFICULTY  LEVEL 

A 

M15.APT 

SA-7 

SA-3  &  SA-10 

SA-9  &  SA-10 

1  - 1  THREAT 

2- 3  THREATS 

3 - 5  THREATS 

B 

M14.APT 

SA-10 

SA-3,  SA-9  &  SA-10 

SA-3  &  SA-7 

1  - 1  THREAT 

2- 3  THREATS 

3- 5  THREATS 

C 

M32.APT 

SA-7 

SA-3  &  SA-10 

SA-3,  SA-7,  SA-9  &  SA-10 

1  -  1  THREAT 

2- 3  THREATS 

3- 5  THREATS 

D 

M12.APT 

SA-3 

SA7,  SA-9 

SA-3,  SA7,  SA-9  &  SA-10 

1  -  1  THREAT 

2  -  3  THREATS 

3-5  THREATS 
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Link  16Page 
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APPENDIX  D 

Defensive  Order  of  Battle  Page 


THIS 

IS  WHERE  THE  MESSAGES 

WILL 

APPEAR . 

LINE 

1 

THIS 

IS  WHERE  THE  MESSAGES 

WILL 

APPEAR. 

LINE 

2 

THIS 

IS  WHERE  THE  MES 

SAGES 

WILL 

APPEAR. 

LINE 

3 

THIS 

IS  WHERE  THE  MES 

SAGES 

WILL 

APPEAR. 

LINE 

/ 

A  I  DOB 

A 

S  N  0  0  0  2 

S  N 

L  A  T  LONG 

SYMBOL  R  A 

MG 

IT 

i„ 

E 

L  A  T 

00  0  1 

N0000.0  E  0  O  0 

00  . 

0 

S  3 

Y 

y 

Y 

N 

r 

LONG 

0  0  0  2 

N00  00.0  E  8  0  0 

8  0  . 

0 

S  3 

Y 

V 

Y 

N 

0  00  3 

N  00  00.0  E  0  0  0 

0  0  . 

0 

S  3 

Y 

V 

Y 

N 

C 

SYMBOL 

0  0  0  4 

N 0000.0  E000 

0  0  . 

0 

S  J 

Y 

Y 

Y 

N 

000  5 

N  00  0  0  .  0  E  0  0  0 

8  0  . 

0 

S  3 

Y 

V 

Y 

N 

r 

7  5  N  M  /  N  0 

0  8  0  G 

N  0  0  0  0.0  E  0  0  8 

0  0  . 

0 

S  3 

Y 

V 

V 

N 

0  00  7 

N0R00.B  E 0  0  0 

0  0  . 

0 

S  8 

Y 

V 

Y 

N 

r 

5  8  N  M  /  S'  0 

00  0  8 

N0000.O  E  8  0  B 

00  . 

0 

3  J 

Y 

v 

Y 

N 

c 

1  0  0  N  M  /  N  0 

200 N M/NO 

SYS 

0  M 

N  A  V 

SMS 

— > 
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APPENDIX  E 


Randomized  Order  of  Presentation 


OSO  TASK  1 

SUBJECT  # 

MODE  OF  PR] 

ESENTATION 

1 

V 

F 

I 

V 

F 

I 

2 

F 

I 

V 

F 

I 

V 

3 

I 

V 

F 

i 

V 

F 

4 

I 

F 

V 

i 

V 

F 

5 

F 

V 

i 

F 

V 

I 

6 

V 

I 

F 

V 

I 

F 

7 

V 

I 

F 

V 

I 

F 

8 

I 

F 

V 

I 

F 

V 

9 

F 

V 

I 

F 

V 

I 

10 

F 

I 

V 

F 

I 

V 

11 

I 

V 

F 

I 

V 

F 

12 

V 

F 

I 

V 

F 

I 

Note:  Subject  #12  did  not  appear  as  scheduled — this  order  was  not  tested 


DSO  TASK 

SUBJECT# 

MODE  OF  PRESENTATION  1 

1 

V 

V 

F 

F 

2 

F 

V 

V 

F 

3 

V 

F 

V 

F 

4 

F 

F 

V 

V 

5 

V 

F 

F 

V 

6 

F 

V 

F 

V 

7 

F 

F 

V 

V 

8 

V 

F 

F 

V 

9 

F 

V 

F 

V 

10 

V 

V 

F 

F 

11 

F 

V 

V 

F 

12 

V 

F 

V 

F 

V- AUDIO  (VOICE) 

F  -  FREETEXT  MESSAGE 
I  -  INTEGRATED 

Note:  Odd  subject  numbers  did  OSO  task  first;  even  subject  numbers  did  DSO  task  first.  Subject  #12  did 
not  appear  when  scheduled — this  order  was  not  tested 
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MEDIANS 


APPENDIX  F 
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MEDIANS  MEDIANS 


APPENDIX  F 


TARGET  LOW  CONDITION 


MODE 

Figure  3.  Comparing  Modes  in  the  OSO  Task:  Median  Strokes,  Time  and  SWAT 


TARGET  LOW  CONDITION 


MODE 

Figure  4.  Comparing  Modes  in  the  OSO  Task:  Median  Errors 
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MEDIANS 


APPENDIX  F 


TARGET  MEDIUM  CONDITION 


MODE 

Figure  5.  Comparing  Modes  in  the  OSO  Task:  Median  Strokes,  Time  and  SWAT 


TARGET  MEDIUM  CONDITION 


Figure  6.  Comparing  Modes  in  the  OSO  Task:  Median  Errors 
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MEDIANS 


APPENIXF 


AUDIO  FREETEXT  INTEGRATED 

MODE 

Figure  8.  Comparing  Modes  in  the  OSO  Task:  Median  Errors 


APPENDIX  F 


TABLE  1.  OVERALL  INDICES  OF  CENTRAL  TENDENCY  AND  DISPERSION 

OSO  TASK 


MODE 


AUDIO 

FREE TEXT 

INTEGRATED 

STROKES 

Mean 

57.817 

61.884 

13.269 

Std  Deviation 

18.727 

18.446 

7.302 

Median 

47.750 

53.250 

10.950 

Percentile  25 

43.450 

47.950 

7.550 

Percentile  75 

77.000 

83.250 

15.750 

Mode 

37.700 

53.250 

7.500 

Range 

61.300 

56.950 

24.500 

Minimum 

:  37.700 

41.300 

6.000 

Maximum 

99.000 

98.250 

30.500 

TIME 


Mean 

96.921 

69.379 

28.889 

Std  Deviation 

32.793 

26.848 

14.128 

Median 

86.879 

60.101 

24.084 

Percentile  25 

70.346 

48.117 

17.991 

Percentile  75 

122.508 

82.998 

41.545 

Mode 

56.404 

37.459 

12.478 

Range 

116.941 

104.666 

44.867 

Minimum 

56.404 

37.459 

12.478 

Maximum 

173.345 

142.125 

57.345 

ERRORS 


Mean 

.193 

.209 

.000 

Std  Deviation 

.347 

.388 

.000 

Median 

.  000 

.000 

.000 

Percentile  25 

.  000 

.000 

.000 

Percentile  75 

.250 

.250 

.000 

Mode 

.000 

.000 

.000 

Range 

1.250 

1.500 

.000 

Minimum 

.000 

.000 

.000 

Maximum 

1.250 

1.500 

.000 

SWAT 


Mean 

26.357 

22.195 

5.542 

Std  Deviation 

23.853 

18.517 

7.370 

Median 

19.400 

16.700 

.000 

Percentile  25 

7.700 

5.400 

.000 

Percentile  75 

46.800 

37.800 

11.550 

Mode 

.000 

.000 

.000 

Range 

74.775 

58.050 

20.950 

Minimum 

.000 

.000 

.000 

Maximum 

74.775 

58.050 

20.950 
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TABLE  2.  OVERALL  CORRELATION  COEFFICIENTS 
OSO  TASK 


STROKES 


TIME 


ERRORS 


TIME 

.8932 
(  99) 

P=  .000 


ERRORS 

.5168 
(  99) 

P=  .000 

.5024 
(  99) 

P=  .000 


SWAT 

.1787 
<  99) 

P=  .077 

.1140 
(  99) 

P=  .261 

.1179 
(  99) 

P=  .245 
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APPENDIX  F 


TABLE  3.  OVERALL  REPEATED  MEASURES  MANOVA 

DEPENDENT  MEASURES:  STROKES,  TIME  AND  ERROR 
OSO  TASK 


Tests  involving  ’MODE’  Within- Subject  Effect. 


EFFECT  . .  MODE 

Multivariate  Tests  of  Significance  (S  =  1,  M  =  2  ,N=1  1/2) 


Test  Name 

Value 

Exact  F 

Hypoth.  DF 

Error  DF 

Sig.  of  F 

Pillais 

.99790 

395.22973 

6.00 

5.00 

.000 

Hotellings 

474.27568 

395.22973 

6.00 

5.00 

.000 

Wilks 

.00210 

395.22973 

6.00 

5.00 

.000 

Roys  .99790 

Note..  F  statistics  are  exact. 


Tests  involving  'TARGET'  Within- Subject  Effect. 
EFFECT  . .  TARGET 


Multivariate  Tests 

of  Significance 

(S  =  1,  M  =  2 

,  N  -  1  1/2) 

Test  Name 

Value 

Exact  F 

Hypoth.  DF 

Error  DF 

Sig.  of  F 

Pillais 

. 99599 

206.96160 

6.00 

5.00 

.000 

Hotellings 

248.35392 

206.96160 

6.00 

5.00 

.000 

Wilks 

.00401 

206.96160 

6.00 

5.00 

.000 

Roys 

.99599 

Note..  F  statistics 

are  exact . 

Tests  involving  'MODE  BY  TARGET'  Within- Subject  Effect. 


EFFECT  . .  MODE  BY  TARGET 

AVERAGED  Multivariate  Tests  of  Significance  (S=3,  M  =  0,  N=18  ) 


Test  Name 

Value 

Approx .  F 

Hypoth .  DF 

Error  DF 

Sig.  of  F 

Pillais 

1.40084 

8.75985 

12.00 

120.00 

.000 

Hotellings 

9.49238 

29.00449 

12.00 

110.00 

.000 

Wilks 

.05314 

17.07553 

12.00 

100.83 

.000 

Roys 

.89525 
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TABLE  4.  OVERALL  REPEATED  MEASURES  MANOVA 
DEPENDENT  MEASURE:  SWAT 
OSO  TASK 


EFFECT  . .  MODE 

Multivariate  Tests  of  Significance  (S  =  1,  M  =  0,  N  =  3  1/2) 


Test  Name 

Value 

Exact  F 

Hypoth.  DF 

Error  DF 

Sig.  of  F 

Pillais 

.61151 

7.08339 

2.00 

9.00 

.014 

Hotellings 

1.57409 

7.08339 

2.00 

9.00 

.014 

Wilks 

.38849 

7.08339 

2.00 

9.00 

.014 

Roys  .61151 

Note. .  F  statistics  are  exact. 


EFFECT  . .  TARGET 

Multivariate  Tests  of  Significance  (S  =  1,  M=0,  N  =  3  1/2) 


Test  Name 

Value 

Exact  F 

Hypoth .  DF 

Error  DF 

Sig.  of  F 

Pillais 

.77213 

15.24839 

2.00 

9.00 

.001 

Hotellings 

3.38853 

15.24839 

2.00 

9.00 

.001 

Wilks 

.22787 

15.24839 

2.00 

9.00 

.001 

Roys  . 77213 

Note..  F  statistics  are  exact. 


EFFECT  . .  MODE  BY  TARGET 


Multivariate  Tests 

of  Significance 

(S  =  1,  M  =  1 

r  N  =  2  1/2) 

Test  Name 

Value 

Exact  F 

Hypoth.  DF 

Error  DF 

Sig.  of  F 

Pillais 

.73962 

4.97087 

4.00 

7.00 

.032 

Hotellings 

2.84049 

4.97087 

4.00 

7.00 

.032 

Wilks 

.26038 

4.97087 

4.00 

7.00 

.032 

Roys 

.73962 

Note..  F  statistics 

are  exact . 
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TABLE  5.  INDICES  OF  CENTRAL  TENDENCY  AND  DISPERSION 
FOR  EACH  TARGET  CONDITION 
OSO  TASK 


TARGET 


i 

' 

LOW 

MEDIUM 

1 

MODE 

Total 

MODE 

Total 

AUDIO 

i  FREE TEXT 

INTEGRATED 

AUDIO 

FREETEXT 

INTEGRATED 

STROKES 

Mean 

81.909 

86.432 

20.727 

63.023 

48.795 

53.470 

10.765 

37.677 

Std  Deviation 

8.405 

5.046 

6.843 

31.154 

5.854 

4 . 613 

4.976 

20.057 

Median 

79.500 

84.250 

17.000 

79.250 

46.917 

51.833 

8.833 

46.917 

Percentile  25 

77.000 

83.250 

15.000 

27.750 

45.250 

50.750 

!  7.500 

11.667 

Percentile  75 

81.500 

89.250 

27.750 

84.000 

53.167 

56.333 

11.667  [ 

51.833 

Mode 

74.500 

83.250 

13.250 

13.250 

46.917 

53.250 

7.5oo  ; 

7.500 

Range 

24.500 

19.000 

17.250 

85.750 

19.000 

13.750 

17.417  ■ 

56.000 

Minimum 

74.500 

79.250 

13.250 

13.250 

42.750 

48.250 

6.000 

6.000 

Maximum 

99.000 

98.250 

30.500 

! 

99.000 

61.750 

62.000 

23.417 

62.000 

TIME 

1 

Mean 

136.526 

99.932 

44.749 

93.736 

83.010 

58.844 

24.536 

55.463 

Std  Deviation 

19.007 

21.062 

9.003 

41.765 

15.338 

12.810 

10.103 

27.391 

Median 

127.060 

91.140 

45.620 

91.140 

78.074 

57.604 

21.877 

57.604 

Percentile  25 

122.508 

82.945 

39.875 

52.703 

68.837 

48.366 

!  17.991 

25.293 

Percentile  75 

154.230 

118.818 

52.703 

126.503 

92.998 

62.754 

25.293 

76.619 

Mode 

112.378 

77.460 

28.838 

28.838 

63.398 

44.280 

16.457 

16.457 

Range 

60.968 

64.665 

28.508 

144.508 

46.839 

47.837 

35.403 

93.781  ! 

Minimum 

112.378 

77.460 

28.838 

28.838 

63.398 

44.280 

16.457  1 

16.457  | 

Maximum 

173.345 

|  142.125 

57.345 

173.345 

110.238 

92.117 

51.859  j 

110.238 

ERRORS 

i 

1 

1  Mean 

.364 

.409 

.000 

.258 

.083 

.159 

.000  i 

.081  | 

Std  Deviation 

.517 

.539 

.000 

.457 

.149 

.322 

.000  ! 

.209  i 

|  Median 

.  000 

|  .250 

.000 

.000 

.000 

.  000 

.ooo  i 

.000  i 

Percentile  25 

.000 

.000 

.000 

.000 

.000 

.  000 

.000 

.000 

|  Percentile  75 

1.000 

.750 

.000 

.250 

.083 

.250 

.000 

.000 

Mode 

.  000 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

Range 

1.250 

1.500 

.000 

1.500 

.417 

1.000 

.000 

1.000 

Minimum 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

Maximum 

1  . . 

1.250 

1.500 

.000 

1.500 

.417 

1.000 

.000  j 

1.000 

|  SWAT 

Mean 

13.339 

11.457 

2.982 

9.259 

27.573 

23.189 

5.727 

18.830 

Std  Deviation 

19.352 

15.558 

5.104 

14.891 

22.572 

15.334 

8.391 

18.615 

Median 

2.700 

2.700 

.000 

.000 

19.400 

28.650 

.000  , 

11.550 

Percentile  25 

.000 

.000 

.000 

.000 

9.250 

9.250 

.000 

2.700  ; 

Percentile  75 

20.600 

16.700 

6.550 

14.650 

46.800 

36.225 

14.400 

33.425 

Mode 

.000 

.000 

.000 

.000 

46.800 

11.550 

.000 

.000 

Range 

1 

52.725 

41.650 

;  15.700 

52.725 

64.575 

44.100 

20.950  . 

64.575  1 

Minimum 

.000 

1  .000 

.000 

.000 

.000 

2.700 

.000 

.000 

Maximum 

52.725 

,  41.650 

15.700 

52.725 

64.575 

46.800 

20.950 

64.575  ! 
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TABLE  5  (CONTINUED) 

TARGET 


HIGH 

MODE  Total 


AUDIO 

FREETEXT 

INTEGRATED 

STROKES 

Mean 

42.745 

45.750 

8.314 

32.270 

Std  Deviation 

6.285 

3.870 

1.987 

17.769 

Median 

40.300 

44.450 

7.550 

40.300 

Percentile  25 

38.500 

42.800 

6.800 

8.800 

Percentile  75 

44.300 

47.950 

8.800 

44.450 

Mode 

37.700 

41.300 

8.800 

8.800 

Range 

20.700 

13.100 

6.200 

51.750 

Minimum 

37.700 

41.300 

6.650 

6.650 

Maximum 

58.400 

54.400 

12.850 

58.400 

TIME 

Mean 

71.228 

49.360 

17.381 

45.990 

Std  Deviation 

13.429 

10.450 

3.523 

24.465 

Median 

70.346 

45.195 

16.578 

45.195 

Percentile  25 

60.666 

42.725 

14.677 

19.040 

Percentile  75 

77.064 

54 . 159 

19.040 

61.213 

Mode 

56.404 

37.459 

12.478 

12.478 

Range 

44.287 

36.097 

11.606 

88.212 

Minimum 

56.404 

37.459 

12.478 

12.478 

Maximum 

100.690 

73.556 

24.084 

100.690 

ERRORS 

Mean 

.132 

.059 

.000 

.  064 

Std  Deviation 

.218 

.122 

.000 

.150 

Median 

.000 

.000 

.000 

.000 

Percentile  25 

.000 

.000 

.000 

.000 

Percentile  75 

.250 

.050 

.000 

.000 

Mode 

.000 

.000 

.000 

.000 

Range 

.700 

.350 

.000 

.700 

Minimum 

.000 

.000 

.000 

.000 

Maximum 

.  700 

.350 

.000 

.700 

SWAT 

Mean 

38.159 

31.939 

7.918 

26.005 

Std  Deviation 

24.400 

19.688 

7.995 

22.416 

Median 

32.125 

26.200 

5.150 

19.550 

Percentile  25 

15.800 

10.800 

.000 

8.100 

Percentile  75 

68.225 

50.475 

15.400 

47.575 

Mode 

7.700 

3.850 

.000 

.000 

Range 

67.075 

54.200 

19.550 

74.775 

Minimum 

7.700 

3.850 

.000 

.  000 

Maximum 

74.775 

58.050 

19.550 

74.775 
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AUDIO  FREETEXT 

MODE 

Figure  1 .  Comparing  Modes  In  the  DSO  Task:  Median  Strokes  and  Time 


.15 

.13 


AUDIO  FREETEXT 

MODE 

Figure  2.  Comparing  Modes  in  the  DSO  Task:  Median  Errors 
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AUDIO  FREETEXT 

MODE 

Figure  4.  Comparing  Modes  in  the  DSO  Task:  Median  Strokes  and  Time 
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TABLE  1.  OVERALL  INDICES  OF  CENTRAL  TENDENCY  AND  DISPERSION 

DSO  TASK 


MODE 


AUDIO 

FREE TEXT 

STROKES 

Mean 

22.68 

25.76 

Std  Deviation 

1.94 

2.58 

Median 

22.20 

25.25 

Percentile  25 

21.25 

23.58 

Percentile  75 

23.67 

26.55 

Mode 

21.00 

23.50 

Range 

8.50 

10.20 

Minimum 

20.50 

21.80 

Maximum 

29.00 

32.00 

ERRORS 

Mean 

.14 

.12 

Std  Deviation 

!  .21 

.17 

Median 

.08 

.08 

Percentile  25 

.00 

.00 

Percentile  75 

.25 

.20 

Mode 

.00 

.00 

Range 

.95 

.83 

Minimum 

.00 

.00 

Maximum 

.  95 

.83 

TIME 

Mean 

39.83 

28.01 

Std  Deviation 

6.03 

6.03 

Median 

39.40 

28.10 

Percentile  25 

36.35 

24.29 

Percentile  75 

43.19 

31.33 

Mode 

27.92 

17.73 

Range 

26.97 

20.87 

Minimum 

27.92 

17.73 

Maximum 

54.89 

38.60 

SWAT 

Mean 

17.33 

10.69 

Std  Deviation 

20.90 

14.80 

Median 

7.70 

2.70 

Percentile  25 

.00 

.00 

Percentile  75 

30.78 

19.65 

Mode 

.00 

.00 

Range 

;  65.80 

54.55 

Minimum 

.00 

.00 

Maximum 

65.80 

54.55 
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TABLE  2 


STROKES 


TIME 


ERRORS 


.  OVERALL  CORRELATION  COEFFICIENTS 
DSO  TASK 


TIME 

- . 1140 
(  66) 
P=  .362 


ERRORS 

.2384 
(  66) 
P=  .054 

.1379 
(  66) 
P=  .270 


SWAT 

-.1285 
(  66) 
P=  .304 

-.1182 
(  66) 
P=  .344 

-.1075 
(  66) 
P=  .390 
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TABLE  3.  OVERALL  REPEATED  MEASURES  MANOVA 
DEPENDENT  MEASURE:  STROKES 
DSO  TASK 


Tests  involving  'MODE'  Wi thin-Subject  Effect. 


[mode  has  only  two  levels — same  as  averaged  univariate  test] 


Source  of  Variation 

SS 

DF 

MS 

F 

Sig  of  F 

WI TH IN+RE S IDUAL 

29.25 

10 

2.93 

MODE 

156.49 

1 

156.49 

53.50 

.000 

EFFECT  . .  THREAT 

Multivariate  Tests 

of  Significance 

(S  =  1,  M  =  0, 

N  =  3  1/2) 

Test  Name 

Value 

Exact  F 

Hypoth .  DF 

Error  DF 

Sig.  of  F 

Pillais 

.19077 

1.06083 

2.00 

o 

o 

CTi 

.386 

Hotellings 

.23574 

1.06083 

2.00 

9.00 

.386 

Wilks 

Roys 

.80923 

.19077 

1.06083 

2.00 

9.00 

.386 

Note. .  F  statistics  are  exact. 


EFFECT  . .  MODE  BY  THREAT 

Multivariate  Tests  of  Significance  <S=  1,  M  =  0,  N=  3  1/2) 


Test  Name 

Value 

Exact  F 

Hypoth .  DF 

Error  DF 

Sig.  of  F 

Pillais 

.38438 

2.80972 

2.00 

9.00 

.113 

Hotellings 

.62438 

2.80972 

2.00 

9.00 

.113 

Wilks 

.61562 

2.80972 

2.00 

9.00 

.113 

Roys 

.38438 

Note. .  F  statistics  are  exact. 
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TABLE  4.  OVERALL  REPEATED  MEASURES  MANOVA 
DEPENDENT  MEASURE:  TIME 
DSO  TASK 


Tests  involving  ’MODE'  Wi thin-Subject  Effect. 

[mode  has  only  two  levels- -same  as  averaged  univariate  test] 


Source  of  Variation 

SS 

DF 

MS 

F 

Sig  of  F 

WITHIN+RESIDUAL 

136.48 

10 

13.65 

MODE 

2305.05 

1 

2305.05 

168.90 

.000 

EFFECT  . .  THREAT 
Multivariate  Tests 

of  Significance 

(S  =  1,  M  =  0, 

N  =  3  1/2) 

Test  Name 

Value 

Exact  F 

Hypoth.  DF 

Error  DF 

Pillais 

.64851 

8.30270 

2.00 

9.00 

Hotellings 

1.84504 

8.30270 

2.00 

9.00 

Wilks 

.35149 

8.30270 

2.00 

9.00 

Roys 

.64851 

Note..  F  statistics 

are  exact. 

EFFECT  . .  MODE  BY  THREAT 
Multivariate  Tests  of  Significance 

(S  =  1 ,  M  =  0 , 

N  =  3  1/2) 

Test  Name 

Value 

Exact  F 

Hypoth.  DF 

Error  DF 

Pillais 

.59609 

6.64097 

2.00 

9.00 

Hotellings 

1.47577 

6.64097 

2.00 

9.00 

Wilks 

Roys 

Note. .  F  statistics 

.40391 
.59609 
are  exact. 

6.64097 

2.00 

9.00 

Sig. 


Sig . 


of  F 

.009 

.009 

.009 


of  F 

.017 

.017 

.017 
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TABLE  5.  OVERALL  REPEATED  MEASURES  MANOVA 
DEPENDENT  MEASURE:  ERRORS 
DSO  TASK 


Tests  involving  'MODE'  Wi thin-Subject  Effect. 

[mode  has  only  two  levels — same  as  averaged  univariate  test] 


Source  of  Variation 

SS 

DF 

MS 

F 

Sig  of  F 

WITHIN+RESIDUAL 

.10 

10 

.01 

MODE 

.01 

1 

.01 

.85 

.378 

EFFECT  . .  THREAT 
Multivariate  Tests  of 

Significance 

II 

M 

M  =  0 ,  N  = 

3  1/2) 

Test  Name 

Value 

Exact  F 

Hypoth.  DF 

Error  DF 

Sig.  of  F 

Pillais 

.29740 

1.90474 

2.00 

9.00 

.204 

Hotellings 

.42328 

1.90474 

2.00 

9.00 

.204 

Wilks 

.70260 

1.90474 

2.00 

9.00 

.204 

Roys 

.29740 

Note..  F  statistics 

are  exact. 

EFFECT  . .  MODE  BY 

THREAT 

Multivariate  Tests 

of  Significance 

(S  =  1 ,  M  =  0 , 

N  =  3  1/2) 

Test  Name 

Value 

Exact  F 

Hypoth.  DF 

Error  DF 

Sig.  of  F 

Pillais 

.28815 

1.82157 

2.00 

9.00 

.217 

Hotellings 

.40479 

1.82157 

2.00 

9.00 

.217 

Wilks 

.71185 

1.82157 

2.00 

9.00 

.217 

Roys 

.28815 

Note..  F  statistics  are  exact. 
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TABLE  6.  OVERALL  REPEATED  MEASURES  MANOVA 
DEPENDENT  MEASURE:  SWAT 
DSO  TASK 


Tests  involving  'MODE'  Within- Subject  Effect. 

[mode  has  only  two  levels-~same  as  averaged  univariate  test] 


Source  of  Variation 

SS 

DF 

MS 

F 

Sig  of  F 

WITHIN+RESIDUAL 

2164.61 

10 

216.46 

MODE 

726.74 

1 

726.74 

3.36 

.097 

EFFECT  .  .  THREAT 

Multivariate  Tests  of  Significance  (S  -  1,  M  =  0,  N  =  3  1/2) 


Test  Name 

Value 

Exact  F 

Hypoth .  DF 

Error  DF 

Sig.  of  F 

Pillais 

.69131 

10.07752 

2.00 

9.00 

.005 

Hotellings 

2.23945 

10.07752 

2.00 

9.00 

.005 

Wilks 

.30869 

10.07752 

2.00 

9.00 

.005 

Roys 

.69131 

Note..  F  statistics 

are  exact. 

EFFECT  . .  MODE  BY 

THREAT 

Multivariate  Tests 

of  Significance 

(S  =  1 ,  M  =  0 , 

N  =  3  1/2) 

Test  Name 

Value 

Exact  F 

Hypoth .  DF 

Error  DF 

Sig.  of  F 

Pillais 

.22734 

1.32406 

2.00 

9.00 

.313 

Hotellings 

.29423 

1.32406 

2.00 

9.00 

.313 

Wilks 

.77266 

1.32406 

2.00 

9.00 

.313 

Roys 

.22734 

Note..  F  statistics  are  exact. 
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TABLE  7. 

INDICES  OF 

CENTRAL  TENDENCY 

AND  DISPERSION 

FOR 

EACH  THREAT  CONDITION 

DSO  TASK 

. . 

THREAT 

LOW 

MEDIUM 

HIGH 

MODE 

Total 

MODE 

Total 

MODE 

AUDIO 

FREETEXT 

AUDIO  FREETEXT 

AUDIO 

FREETEXT 

STROKES 

Mean 

23.09 

25.27 

24 . 18 

22.15 

25.47 

23.81 

22.79 

26.53 

Std  Deviation 

2.30 

2.44 

2.57 

1.51 

2.81 

2.78 

1.99 

2.54 

Median 

22.75 

24.75 

23.63 

21.83 

24.11 

23.29 

22.20 

26.55 

Percentile  25 

21.50 

23.50 

22.75 

21.00 

23.42 

21.83 

i  21.05 

24.85 

Percentile  75 

24 . 00 

25.50 

25.25 

23.17 

26.33 

25.50 

,  24.65 

28.75 

Mode 

21.00 

23.50 

23.50 

20.50 

22.83 

22.83 

20.55 

26.55 

Range 

8.00 

8.50 

11.00 

5.00 

8.25 

10.58 

5.35 

8.05 

Minimum 

21.00 

23.50 

21.00 

20.50 

22.83 

20.50 

20.55 

21.80 

Maximum 

j  29.00 

i  . 

32.00 

32.00 

25.50 

31.08 

31.08 

25.90 

29.85 

TIME 

Mean 

44 . 01 

28.67 

36.34 

36.78 

27.46 

32.12 

38.70 

27.91 

Std  Deviation 

6.30 

5.33 

9.70 

4.27 

6.37  ! 

7.13 

5.26 

6.81 

Median 

43.10 

28.92 

36.32 

37.40 

27.95 

33.27 

38.24 

27.49 

;  Percentile  25 

39.68 

24.81 

28.92 

33.03 

21.92 

27.95 

36.35 

23.72 

Percentile  75 

j  49.82 

30.65 

1  43.10 

39.62 

33.52 

37.40 

i  43.19 

34.05 

Mode 

35.50 

19.02 

19.02 

28.42 

18.01 

18.01 

|  27.92 

17.73 

[  Range 

19.39 

19.42 

35.87 

15.26 

18.51 

25.67 

18.90 

20.87 

|  Minimum 

35.50 

19.02 

19.02 

28.42 

18.01  I 

18.01 

27.92 

17.73 

j  Maximum 

54.89 

38.44 

54.89 

43.68 

36.52 

43.68 

46.82 

38.60 

ERRORS 

i 

Mean 

.07 

.07 

.07 

.11 

.19 

.15 

i  26 

.11 

1  Std  Deviation 

.  16 

.12 

.14 

.12 

.25 

.20 

.27 

.09 

Median 

.00 

.00 

.00 

.08 

.08  1 

.08 

!  .20 

.15 

Percentile  25 

.  00 

.00 

.00 

.00 

.00 

.00 

.  10 

.00 

Percentile  75 

1  .00 

:  .25 

.00 

.25 

.  33 

.25 

.40 

;  .20 

Mode 

.00 

.00 

.00 

.00 

.08 

.08 

.00 

.00 

Range 

.50 

.25 

.50 

.33  | 

.83 

.83 

.  95 

i  -25 

Minimum 

.00 

.00 

.00 

.00  ; 

.00 

.00 

.00 

.00 

Maximum 

.50 

.25 

.50 

.33 

.83 

.  83 

.  95 

1  .25 

j  SWAT 

!  Mean 

5.59 

.49 

3.04 

16.69  ' 

11.74 

14.21 

29.71 

19.84 

I  Std  Deviation 

12.67 

1.09 

9.15 

20.16 

15.91 

17.90 

22.59 

15.46 

I  Median 

.00 

.00 

.00 

7.85 

2.70 

6.63 

29.93 

19.65 

Percentile  25 

.00 

.00 

.00 

2.70 

.00 

.00 

10.40 

5.15 

Percentile  75 

5.40 

.00 

2.70 

30.78 

22.25 

22.25 

47.75 

28.65 

|  Mode 

.00 

.00 

.00 

.00 

.00 

.00 

.00 

.00 

i  Range 

42.95 

2.70 

42. 95 

i  58.43 

46.80 

58.43 

65.80 

54.55 

Minimum 

.00 

1  .00 

.00 

.00 

.00 

.00 

I  .00 

.00 

Maximum 

42. 95 

2.70 

42.95 

58.43 

46.80 

58.43 

:  65.80 

54.55 
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TABLE  7  (CONTINUED) 

THREAT 

HIGH 

Total 

STROKES 

Mean  24.66 

Std  Deviation  2 . 94 

Median  24 . 75 

Percentile  25  21 . 80 

Percentile  75  26.55 

Mode  26.55 

Range  9 . 30 

Minimum  20.55 

Maximum  29.85 

TIME 

Mean  33 . 31 

Std  Deviation  8.11 

Median  35 . 21 

Percentile  25  27.49 

Percentile  75  38.60 

Mode  17.73 

Range  29.08 

Minimum  17.73 

Maximum  46.82 

ERRORS 

Mean  .18 

Std  Deviation  .21 

Median  .15 

Percentile  25  .00 

Percentile  75  .20 

Mode  . 00 

Range  . 95 

Minimum  .00 

Maximum  .  95 

SWAT 

Mean  24.78 

Std  Deviation  19.55 

Median  23.66 

Percentile  25  10.40 

Percentile  75  33.80 

Mode  . 00 

Range  65.80 

Minimum  . 00 

Maximum  65 . 80 
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